Editing conventional or light-field images to provide effects, such as changing colorization, changing contrast, or inserting and/or removing objects in the image, can be challenging. Typically, the user must employ careful selection of object boundaries to control how the effects are applied. Accordingly, application of depth-based effects can be a time-consuming and labor-intensive effort.
A further challenge is presented by the need to make depth-based modifications, such as background replacement, to video. The process of drawing a distinction between foreground and background elements can rapidly grow cumbersome when multiple frames are involved. Known methods for automating such segmentation are significantly limited. For example, edge detection and alpha estimation in the edge regions rely on separating background and foreground colors, which is inaccurate in low contrast areas or where the foreground and background colors are similar.
The challenge is compounded in the case of video with multiple viewpoints, as with a light-field camera or a tiled camera array. If segmentation is required for each view in the video stream, then the process must be repeated accordingly. The result is a very labor-intensive process.